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DETAILED ACTION 

Continued Examination Under 37 CFR 1. 1 14 

1 . A request for continued examination under 37 CFR 1.114, including tine fee set 
forth in 37 CFR 1 .17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1 .17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
07/06/2010 has been entered. 

Response to Arguments 

2. Applicant's arguments with respect to the prior art rejections have been 
considered but are moot in view of the new ground(s) of rejection. 

Ciaim Objections 

3. Claim 2 is objected to because of the following informalities: The phrase "the 
step of excluding certain of said sub-part motion vectors" is grammatically incorrect. 
Appropriate correction is required. 

Ciaim Rejections - 35 USC § 101 

4. 35 U.S.C. 101 reads as follows: 



Whoever invents or discovers any new and useful process, macliine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 
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5. Claim 32 is rejected under 35 U.S.C. 101 because tiie claimed invention is 
directed to non-statutory subject matter. Claim 32 can be implemented in the form of a 
carrier wave. Examiner suggests adding the limitation "non-transitory" to the existing 
"computer-readable storage medium". 

Claim Rejections - 35 USC §112 

6. The following is a quotation of the first paragraph of 35 U.S.C. 1 12: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

7. Claims 1 -8 and 1 0-32 are rejected under 35 U.S.C. 1 1 2, first paragraph, as 
failing to comply with the written description requirement. The claim(s) contains subject 
matter which was not described in the specification in such a way as to reasonably 
convey to one skilled in the relevant art that the inventor(s), at the time the application 
was filed, had possession of the claimed invention. Claims 1-5, 8, 10- 12, 17-22 and 
24-26 recite motion vectors for sub-parts of a frame as well as global motions for 
multiple entire frames. Applicant's specification is silent regarding these claim 
limitations. The claims dependent from these claims inherit this deficiency. 

8. The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 
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9. Claims 1 3-16 are rejected under 35 U.S.C. 1 1 2, second paragrapli, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 

1 0. Claims 1 3-16 recite the limitation "the selection step". It is unclear which 
selection step is being referred to as multiple selection steps have been previously set 
forth. 

1 1 . The following is a quotation of the fourth paragraph of 35 U.S.C. 112: 

[A] claim in dependent form shall contain a reference to a claim previously set forth and then specify a 
further limitation of the subject matter claimed. A claim in dependent form shall be construed to 
incorporate by reference all the limitations of the claim to which it refers. 

1 2. Claim 32 is rejected under 35 U.S.C. 1 1 2, fourth paragraph, for failing to further 
limit the subject matter claimed. Claim 32 is only for a computer-readable medium, and 
does not require the machine present in claim 1 . 

Claim Rejections - 35 USC § 103 

1 3. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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1 4. Claims 1 -5, 1 1 -1 4, 1 7-21 , 26-29 and 32 are rejected under 35 U.S.C. 1 03(a) as 
being unpatentable over Sun (US Patent Application Publication # 2003/0072373 A1) in 
view of Meer et al. (Meer et al., "Robust Regression Methods for Computer Vision: A 
Review", International Journal of Computer Vision, 6(1):59-70, 1991) and Jinzenji et al. 
(US Patent # 6,977,664 B1). 

15. Regarding claims 1 and 17, 
Sun teaches: 

A method of global motion estimation between frames of a motion-compensated 
inter-frame encoded video sequence, each frame of the sequence having a plurality of 
motion vectors encoded therein relating sub-parts of one frame to sub-parts of a 
preceding and/or succeeding frame of the sequence; the method comprising: 

using digital signal processing circuits including a computer-readable memory 
circuit connected to receive said encoded video sequence and configured to process 
said sequence, for a particular inter-frame encoded frame (Sun: [0028]; [0077]), by: 

a) decoding the motion vectors of the frame (Sun: [0028]: using motion vectors 
from previous coded P frame In current frame); 

b) selecting N sets of the sub-part motion vectors, wherein N is an Integer value 
greater than 1 (Sun: [0057], [0059]-[0060]: motion vectors for image blocks 82, which 
include multiple subsets); 
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c) calculating a motion estimation for each set of sub-part motion vectors (Sun: 
[0027], [0047], [0057]: calculating global motion parameter/vector); 

d) calculating a TSAD value for each motion estimation (Sun: [0057]: TSAD for 
each global motion parameter/vector); 

e) selecting the motion estimation with the least TSAD as that representative of 
the global motion of the entire frame with respect to a preceding or succeeding entire 
anchor frame (Sun: [0057]: the global motion parameter/vector with the lowest TSAD is 
selected as the global motion estimation for the current frame); and 

f) storing or outputting said selected motion estimation (Sun: [0073]). 

Sun fails to teach: 

using a median squared error value; and 

f) storing or outputting said selected motion estimation for use in registering the 
image of said frame with the image of said anchor frame. 

Meer teaches: 

using a median squared error vaiue (Meer: pg 62, right col); 

At the time of invention, it would have been obvious to a person having ordinary 
skill in the art to combine the teachings of Meer with Sun. Both Meer and Sun teach 
trying to lessen the influence of outliers (Meer: pg 62, right col; Sun: [0047]). The 
combination of Meer with Sun is the application of a known technique, lessening the 
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influence of outliers using a median squared error value, to a known device ready for 
improvement, the Sun device, to yield predictable results. 

Jinzenji teaches: 

f) storing or outputting said selected motion estimation for use in registering the 
image of said frame with the image of said anchor frame (Jinzenji: Fig 3: SI ; Fig 4: 1 ; 
col 8, line 29 - col 9, line 16: output of global motion for use in frame mapping to a 
reference coordinate system). 

At the time of invention, it would have been obvious to a person having ordinary 
skill in the art to combine the teachings of Jinzenji with Sun in view of Meer. Outputting 
the global motion for use in image registration, as in Jinzenji, would allow for creation of 
a panoramic image based on the global motion of Sun in view of Meer, thereby allowing 
a user to implement an established use for global motion. 

1 6. Regarding claims 2 and 1 8, 

Sun in view of Meer and Jinzenji teaches: 

further comprising the step of excluding certain of said sub-part motion vectors 
with predetermined characteristics from being selected as a member of one of the N 
sets (Sun: [0059]). 



1 7. Regarding claims 3 and 1 9, 



Application/Control Number: 10/535,621 Page 8 

Art Unit: 2485 

Sun in view of Meer and Jinzenji teaches: 

wherein the excluded sub-part motion vectors include those motion vectors from 
one or more areas substantially around the boundary of the frame (Sun: Fig 8). 

1 8. Regarding claims 4 and 20, 

Sun in view of Meer and Jinzenji teaches: 

wherein the excluded sub-part motion vectors include those motion vectors 
whose value is substantially zero (Sun: [0062]-[0063]). 

19. Regarding claims 5 and 21, 

Sun in view of Meer and Jinzenji teaches: 

A method according to claim 1 , wherein the selecting step b) further comprises 
randomly selecting s sub-part motion vectors from the available sub-part motion vectors 
for each of the N sets (Meer: pg 64, left col: Subsets of the data are chosen by random 
sampling) wherein s is the minimum number for sufficiently estimating a geometrical 
transformation (Meer: pg 64, left col: The points within some error tolerance are called 
the consensus set of the model. If the cardinality, number of points, of the consensus 
set exceeds a threshold, the model is accepted and its parameters recomputed based 
on the whole consensus set. By using a threshold, a minimum number of motion 
vectors must be used. The error tolerance and the consensus set acceptance threshold 
must be set a priori, and can be set according to any minimum level, including the 
minimum number for sufficiently estimating a geometrical transformation.). 
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20. Regarding claim 1 1 , Sun in view of IVIeer and Jinzenji teaclies tine metliod 
limitations of this claim as discussed above with respect to claim 1 . It is noted that 
Jinzenji teaches, and Sun does not teach, the additional step of generating at least one 
panoramic image representing the frames of the video sequence using the global 
motion estimations thus determined (Jinzenji col 8, line 32-56; a provisional sprite 
<panoramic image> is generated). At the time of invention, it would have been obvious 
to a person having ordinary skill in the art to combine the teachings of Jinzenji with Sun 
in view of Meer. Using the panoramic generating means of Jinzenji would allow for 
creation of a panoramic image based on the global motion of Sun in view of Meer, 
thereby allowing a user to implement an established use for global motion. 

21 . Regarding claim 1 2, Sun in view of Meer and Jinzenji teaches: 

A method according to claim 1 1 , wherein the generating step further comprises: 

selecting a particular frame of the sequence as a reference frame, the plane of 

the reference frame being a reference plane (Jinzenji col 8, line 32-56; the reference 

coordinate system which is for the reference frame); 

for each frame other than the reference frame, accumulating the global motion 

estimations from each entire frame back to the entire reference frame (Jinzenji col 8, 

line 32-56; each original image of the arbitrary frames is mapped to a reference 

coordinate system which is for the reference frame); 
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warping each frame otiier tlian tine reference frame onto tfie reference plane 
using the accumulated global motion estimations to give one or more pixel values for 
each pixel position in the reference plane (Jinzenji col 8, line 32-56; frame warping 
occurs when frames are mapped to the reference coordinate system using global 
motion so as to insert or overwrite pixels; and 

for each pixel position in the reference plane, selecting one of the available pixel 
values for use as the pixel value in the panoramic image (Jinzenji col 8, line 32-56; a 
pixel value of a point is obtained from pixel values which exist in the same point). 

22. Regarding claim 1 3, Sun in view of Meer and Jinzenji teaches: 

A method according to claim 12, wherein the selecting step comprises selecting 
a substantially median pixel value from the available pixel values for use in a 
background panoramic image (Jinzenji col 10, line 8-1 1 ; for a plurality of pixels which 
are mapped to the same coordinates, a median value of the pixels is selected as the 
value of the coordinates of the provisional sprite). 

23. Regarding claim 14, Sun in view of Meer and Jinzenji teaches: 

A method according to claim 12, wherein the selecting step comprises selecting 
a substantially most different pixel value from the available pixel values for use in a 
foreground panoramic image (Jinzenji col 8, line 47-51 ; using a threshold to select the 
most different pixel). 
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24. Regarding claim 26, Sun in view of Meer and Jinzenji teaches: 

A system for generating panoramic images from a motion-compensated inter-frame 
encoded video sequence, comprising: 

a system for global motion estimation between frames of a motion-compensated 
inter-frame encoded video sequence as claimed in claim 17 (as shown above), and 
further arranged to provide global motion estimations for each frame (Oh: pg 9, lines 13- 
19); and 

panoramic image generating means for generating at least one panoramic image 

representing the frames of the video sequence using the global motion estimations thus 
determined (Jinzenji col 8, line 32-56; a provisional sprite <panoramic image> is 
generated). 

25. Regarding claim 27, Sun in view of Meer and Jinzenji teaches: 
wherein the panoramic image generating means is further arranged in use to: 

select a particular frame of the sequence as a reference frame, the plane of the 
reference frame thereby being a reference plane (Jinzenji col 8, line 32-56; the 
reference coordinate system which is for the reference frame); 

for each frame other than the reference frame, accumulate the global motion 
estimations from each frame back to the reference frame (Jinzenji col 8, line 32-56; 
each original image of the arbitrary frames is mapped to a reference coordinate system 
which is for the reference frame); 
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warp each frame other than the reference frame onto the reference plane using 
the accumulated global motion estimations to give one or more pixel values for each 
pixel in the reference plane (Jinzenji col 8, line 32-56; frame warping occurs when 
frames are mapped to the reference coordinate system using global motion so as to 
insert or overwrite pixels); and 

for each pixel position in the reference plane, select one of the available pixel 
values for use as the pixel value in the panoramic image (Jinzenji col 8, line 32-56; a 
pixel value of a point is obtained from pixel values which exist in the same point). 

26. Regarding claim 28, Sun in view of Meer and Jinzenji teaches: 

wherein the panoramic image generating means is further arranged to select a 
substantially median pixel value from the available pixel values for use in a background 
panoramic image (Jinzenji col 10, line 8-1 1 ; for a plurality of pixels which are mapped to 
the same coordinates, a median value of the pixels is selected as the value of the 
coordinates of the provisional sprite). 

27. Regarding claim 29, Sun in view of Meer and Jinzenji teaches: 

wherein the panoramic image generating means is further arranged to select a 
substantially most different pixel value from the available pixel values for use in a 
foreground panoramic image (Jinzenji col 8, line 47-51 ; using a threshold to select the 
most different pixel). 
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28. Regarding claim 32, 

Sun in view of IVIeer and Jinzenji teaches: 

A computer-readable storage medium containing a computer program or suite of 
programs arranged such that when executed on a computer system the program or 
suite of programs causes the computer system to perform the method of claim 1 (Sun: 
[0076]-[0077]). 

29. Claims 6-8 and 22-24 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Sun (US Patent Application Publication # 2003/0072373 A1) in view of Meer et al. 
(Meer et al., "Robust Regression Methods for Computer Vision: A Review", International 
Journal of Computer Vision, 6(1):59-70, 1991), Jinzenji et al. (US Patent # 6,977,664 
B1) and Subramaniyan et al. (US Patent Application Publication # 2004/0028134 Al). 

30. Regarding claims 6 and 22, 

Sun in view of Meer and Jinzenji teaches: 

A method according to claim 1 (as shown above) 

Sun in view of Meer and Jinzenji fails to teach: 

f) testing the selected motion estimation representative of the global motion ; and 

g) outputting the selected motion estimation as being representative of the global 
motion of the frame if the test is passed. 
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Subramaniyan teaches: 

the steps of: 

f) testing the selected motion estimation representative of the global motion 
(Subramaniyan: [0035], comparing the difference metric for each of the final motion 
vectors in the previous frame with a predetermined threshold; [0049], This process 
continues until one of 3 conditions are met: [0050], CONDITION 1 : The MSAD Is below 
a threshold THRESH4 . given by: THRESH4=A*Q+B); and 

g) outputting the selected motion estimation as being representative of the global 
motion of the frame If the test Is passed (Subramaniyan: [0035], determining the global 
motion vector based on the each of the final motion vectors in a previous frame with a 
difference metric that is below the threshold; [0049], This process continues until one of 
3 conditions are met: [0050], CONDITION 1 : The MSAD is below a threshold 
THRESH4 . given by: THRESH4=A*Q+B). 

At the time of invention, it would have been obvious to a person having ordinary 
skill in the art to combine the teachings of Subramaniyan with Sun in view of Meer and 

JInzenjI. Testing the global motion estimation before outputting would Improve the Sun 
In view of Meer and JInzenjI method and apparatus by achieving a motion estimation 
that Is high quality while remaining low in complexity (Subramaniyan: [0023]). 

31 . Regarding claims 7 and 23, 

Sun in view of Meer, Jinzenji and Subramaniyan teaches: 
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wherein the test comprises comparing the selected motion estimation with a 
threshold value (Subramaniyan: [0049]-[0050], THRESH4), wherein the test is passed if 
the parameters do not exceed the threshold value (Subramaniyan: [0049]-[0050], [0054] 
The best MV at the end of the second stage is chosen as the best MV for the 
macroblock; the MV having a MSAD below THRESH4 is chosen as the best MV). 

32. Regarding claims 8 and 24, 

Sun in view of Meer, Jinzenji and Subramaniyan teaches: 

wherein if the test is failed, the method further comprises: 

h) determining a motion estimation representative of the global motion of the 
frame with respect to a preceding or succeeding other frame (Subramaniyan: [0035]; 
[0026], motion estimation is computed for blocks of image data from a current image 
frame using one or more previously processed image frames ): 

i) determining a motion estimation representative of the global motion of the other 
frame with respect to the anchor frame ([0035]; the motion estimation circuit 1 10 can 
determine the global motion vector by using an average of all final motion vectors in a 
previous frame ): and 

j) accumulating the motion estimations to give an overall motion estimation 
substantially representative of the global motion of the frame with respect to the anchor 
frame (Subramaniyan: [0035], the motion estimation circuit 1 10 can determine the 
global motion vector by using an average of all final motion vectors in a previous frame). 



Application/Control Number: 10/535,621 Page 16 

Art Unit: 2485 

33. Claims 1 0 and 25 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Sun (US Patent Application Publication # 2003/0072373 A1) in view of Meer et al. 
(Meer et al., "Robust Regression Methods for Computer Vision: A Review", International 
Journal of Computer Vision, 6(1):59-70, 1991), Jinzenji et al. (US Patent # 6,977,664 
B1) and Subramaniyan et al. (US Patent Application Publication # 2004/0028134 Al) 
and Lee et al. (US Patent Application Publication # 2003/0103568 A1). 

34. Regarding claims 1 0 and 25, 

Sun in view of Meer, Jinzenji and Subramaniyan teaches: 
A method according to claim 6 (as shown above) 

Sun in view of Meer, Jinzenji and Subramaniyan fails to teach: 
wherein if the test is failed, the method further comprises: 
interpolating between the motion estimations of adjacent frames to give an 

interpolated motion estimation which is then output as the motion estimation 

representative of the global motion of the frame. 

Lee teaches: 

wherein if the test is failed, the method further comprises: 
interpolating between the motion estimations of adjacent frames to give an 
interpolated motion estimation which is then output as the motion estimation 
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representative of the global motion of the frame (Lee: [0061]-[0063]). 

At the time of invention, it would have been obvious to a person having ordinary 
skill in the art to combine the teachings of Lee with Sun in view of Meer, Jinzenji and 
Subramaniyan. The teachings of Lee provide for motion compensated interpolation that 
eliminates blocking artifacts (Lee [0048]), thereby increasing the ability of the Sun in 
view of Meer, Jinzenji and Subramaniyan teachings to generate accurate global motion 
in the presence of blocking artifacts. 

35. Claims 1 5-1 6 and 30-31 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Sun (US Patent Application Publication # 2003/0072373 A1) in view 
of Meer et al. (Meer et al., "Robust Regression Methods for Computer Vision: A 

Review", International Journal of Computer Vision, 6(1):59-70, 1991), Jinzenji et al. (US 
Patent # 6,977,664 B1) and Szeliski et al. (US Patent # 6,348,918 B1). 

36. Regarding claims 1 5-1 6 and 30-31 , 

Sun in view of Meer and Jinzenji teaches: 

A method according to claim 12 (as shown above). 

Sun in view of Meer and Jinzenji fails to teach: 
wherein the selecting step comprises: 

calculating the mean pixel value of the available pixel values; 
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calculating the LI distance between each available pixel value and the calculated 
mean pixel value; and 

select the pixel value with the median LI distance for use in a background 
panoramic image. 

select the pixel value with the maximum LI distance for use in a foreground 
panoramic image. 

Szeliski teaches: 

wherein the selecting step comprises: 

calculating the mean pixel value of the available pixel values (Szeliski col 8, line 
57-65; taking the mean of the color or intensity values); 

calculating the LI distance between each available pixel value and the calculated 
mean pixel value (Szeliski col 8, line 57-65; where the averaging is weighted by the 
distance of each pixel from the nearest invisible pixel); and 

select the pixel value with the median LI distance for use in a background 
panoramic image (Szeliski col 8, line 57-65; using the median technique). 

select the pixel value with the maximum LI distance for use in a foreground 
panoramic image (Szeliski col 8, line 57-65; the simplest technique is the median 
technique, but many others exist This portion of Szeliski discloses blending specifically 
for a background image. This portion of Szeliski also discloses blending generally. 
Instead of using the median technique for blending background pixels, a maximum 
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technique is obvious for blending foreground pixels. This is because foreground pixels 
are most different from background pixels). 



It would have been obvious to a person having ordinary skill in the art to combine 
the teachings of Szeliski with Sun in view of Meer and Jinzenji. Using the blending 
technique of Szeliski would smooth out disparities of the panoramic image of the Sun in 
view of Meer and Jinzenji teachings, thus creating a panoramic image with increased 
image quality (Szeliski col 9, line 6-8). 



Conclusion 

37. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JAMES PONTIUS whose telephone number is 
(571 )270-7687. The examiner can normally be reached on Monday - Thursday, 8 AM - 
4 PM est.. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jayanti Patel can be reached on (571 ) 272-2988. The fax phone number for 
the organization where this application or proceeding is assigned is 571 -273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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